1
Reporting and Learning from Health IT-Related Events
Toward Safer Healthcare
Session 166, Wednesday, February 13, 2019
Yang Gong, MD, PhD, UTHealth
2
Yang Gong, MD, PhD
Has no real or apparent conflicts of interest to report.
Conflict of Interest
3
Introduction (~5 minutes)
Moderator
Speaker
Presentation (~45 minutes)
Scan QD code for details
Q&A (~10 minutes)
Extended discussion (Posterior to the session)
Twitter @gngyng
LinkedIn GONGYANG@gmail.com
Agenda
4
State the benefits and risks of HIT for patient
safety/healthcare quality improvement
Identify the barriers of reporting and analyzing HIT events
and challenges for turning HIT event reports into actionable
knowledge
Discuss how data representation and knowledge
management in FDA MAUDE incident reports can facilitate
quality improvement towards a better and safer healthcare
system
Learning Objectives
5
Patient Safety
- Pressures and Incentives
Medical error. (Makary & Daniel, 2016)
Leveraging patient safety research: Fifteen-year
efforts since “To Err Is Human” (Liang, Miao, Kang… &
Gong, submitted manuscript, 2018)
6
Individual
Reports
Individual
Reports
Clinicians
Patients
Aggregated
Data
Aggregated
Data
Hospitals
PSO
AnalysisAnalysis
Hospitals
PSO
Actionable
Knowledge
Actionable
Knowledge
Clinicians
Hospitals
Resolving
Problems
Resolving
Problems
Clinicians
Hospitals
Improving Safety through Reporting
Patient Safety Event (PSE)
Reporting Initiatives
To Err is Human (2000)
Patient Safety and Quality
Improvement Act (2005)
Patient Safety Organizations (PSO)
Why Reporting
Learn from lessons
Reporting Mechanism
States working with PSO by 2017
7
Perceived Barriers
Lack of instructions and training
Unsatisfactory usability of
classifications/taxonomies
Lengthy reporting forms
Time-consuming
Competing with other priorities
Lack of motives
No feedback
Observed event seemed “trivial”
» A trivial tip --> a large ‘iceberg’ under water
8
Challenges in Event Reporting
Structured data vs. narratives
Structured data: standardized but limited representations
Narratives: flexible, content-rich, causal and temporal
information
Complexity of medication error reports
8
Multi-Stage
Multi-Personnel
Multi-Factor
Reporting
Analysis
Actions
Learning
Barriers
9
Problem in Reporting
Low Reporting Quality
Data Type
Ideal
ity
Reality
Structured data
Easy to
interpret
Rarely
answered
Unstructured data
Supplementary Info.
Essential Info.
10
Problem in Reporting Errors
- Lacking Analysis Tools
A New Report: “RN removed 100mcg fentanyl from the
omnicell in endo room 2 documented on the anesthesia
sheet that he gave 25mcg, no waste recorded. 75mcg
fentanly unaccounted for.”
Historical Reports & Solutions
Reported, so what?
11
Institute of Medicine (IOM)
HIT plays an imperative role in saving healthcare cost, improving patient outcomes,
decreasing occurrence of medication errors, and refining healthcare process measures
across diverse settings
Agency for Healthcare Research and Quality (AHRQ)
use of information and communication technology in healthcare to support the delivery of
patient or population care or to support patient self-management
Synonym
electronic health records (EHR) and
EHR components
computerized provider order entry (CPOE) or
clinical decision support system (CDSS)
Generalized Health IT also includes
administrative or practice management systems
automated dispensing systems
laboratory information systems, and
diagnostic imaging systems
Nowadays, Health IT has become an integral part of healthcare and has been widely applied to
collect, transmit, process, display, and store patient data
What is Health IT
12
Listed in the top 10 technology-related hazards
new uncertainties and risks for patient safety
disrupting established work patterns
creating new risks in practice, and
encouraging workarounds
The adoption of EHR has revealed potential safety
implications related to
EHR design, implementation, and use related to
technological features of EHR
users and workflows
organizations, rules and regulations
Challenges of Health IT
13
Understand and manage the risks
Sociotechnical context
Sittig and Singh 8-dimensional sociotechnical model
Tool in patient safety studies
Complexities of technology
Users in workflow and external or organizational policies
Health IT in event reporting
Citied as one of contributing factors in reporting systems
No health IT exclusive sources to patient safety studies
AHRQ Common Formats (CFs): Common definitions and
reporting formats (QD code linked)
Hospitals, Community pharmacies, Nursing homes
Meeting the Challenges
14
Goal
Develop a user-centered, knowledge-
based reporting and learning system
Help healthcare practitioners better report events
Connect with relevant reports
Learn how to address causes of errors
Improve the behavior at work
15
Method Filter on Structured Data
4,947,220 reports from
MAUDE 2008-2016
Keyword searching on
Generic Name and
Manufacturer Name
Identify HIT-related events
from sample reports through
domain expert review
6 inclusion criteria
4 exclusion criteria
Assess reviewer
consistency by Cohen’s
kappa
Workflow of reviewing a report from FDA MAUDE database
16
Method
Classifier on Unstructured Data
Term frequency (TF) inverse document frequency (IDF)
Biterm topic model
Extract the semantic themes (topics) from a corpus of short
documents
Classifiers
Random forest, Logistic regression, Naïve Bayes, SVM, J48, JRip
Gold standard:
HIT-related and non-HIT event reports identified by domain experts
17
Result
Keyword-based HIT Event Filter
Generic Names Manufacturer Names Total
Inclusion Keywords 94 38 132
Exclusion Keywords 21 0 21
Year
Raw reports Filtered reports
Reports
HCFA/Manufactur
er/Distributor
Reports
HCFA/Manufactur
er/Distributor
2008
145,598
9,148
1,817
146
2009
201,996
9,906
2,640
214
2010
327,961
10,792
3,434
316
2011
414,083
12,597
2,371
307
2012
520,043
12,952
4,825
308
2013
636,145
12,516
3,551
313
2014
867,451
12,927
4,338
380
2015
965,240
15,762
17,963
384
2016
868,703
15,023
4,685
408
Sum
4,947,220
45,624
Inclusion & exclusion keywords
Raw and filtered reports of 2008-2016 MAUDE database
18
Result
Contributing Factor Distribution Analysis
19
Result
- FDA Medical Device Event Reports
FDA Manufacturer and User Facility
Device Experience (MAUDE) database
~ 6,000,000 events involving medical
devices
structured & unstructured data fields
0.46% ~0.69% are HIT-related events
Up to 50,000 HIT-related events
(QD code linked)
Trend of MAUDE reports (bars) and MAUDE
related publications (line) since 2000
Filter on Structured
data
Classifier on
Unstructured data
Raw Data
HIT
20
Result
An Integrated Model of HIT event Identification
0.4~0.9% 50% 97%
Proportion of
HIT events
Filter on
Structured
data
Classifier on
Unstructured
data
Raw Data HIT
Trade off between precision and recall
Grow the HIT database
21
Underway
Prototype An Integrated Reporting and Shared
Learning System in Healthcare Community
22
Predictive Text Entry
To support reporting
Cueing list, auto-
suggestion
By two-group
randomized test
Improved text
generation
Improved data
consistency and
quality
Entered and
tagged-in text
Initial letters
of input
Auto-suggestion:
matched text
entry hits
(# of hits <=10)
Narrative data entry field equipped with text prediction functions
E
F
G
C
B
Main component lists multiple-choice questions in slide-in mode
Cueing list that reminds
of the content or content
categories of reportable data
A
D
C
Structured Data Entry 13 MCQs and four of them have narrative fields as illustrated as the part B
Unstructured Data Entry One narrative comment field
C: Cueing List
aids in data entry of
specified single-text
field
(B) in the structured
question, or
comment field
G: Auto-suggestion
Suggesting the words, phrases
and sentence in the context to
describe the event details
23
Managing PSE Knowledge
Ontology
Interoperability among
home-grown systems
patient safety organization
(PSO) systems
Data integration
Organizing prevailing
classifications
Decision making
24
Data Source & Annotation
One year data from a PSO
institute (2016)
2,576 medication error
reports (including
adverse drug reaction,
ADR)
Incidents & near misses
Unstructured data
Manual annotating
Two patient safety
domain experts with
pharmacy or clinical
backgrounds
Followed the NCC
MERP taxonomy for
medication errors, plus
ADR
Error
Stages
(6)
Error
Stages
(6)
Error
Types
(8)
Error
Types
(8)
Error
Causes
(5)
Error
Causes
(5)
Administering
Dispensing
Med. Rec.
Monitoring
Ordering
Transcribing
ADR
Billing Issue
Missing Dose
Wrong Admin.
Wrong Document.
Wrong Dose
Wrong Drug
Wrong Time
Devices (HIT)
Information Deficit
Pathophysiological Factor
Performance Deficit
Others
PSO Reports
25
Pipeline
Identifying Three Key Factors
Remove punctuation
Remove number
Rainbow stop word
N-grams tokenizer
TF-IDF
26
Pipeline
Similarity Measurement
Raw Reports
Similarity
Calculation
Grouped Reports
Labels:
Originating stage
Type
Cause
27
Data Elements in Medication Error
28
Leveraging Patient Safety
Information models act as the core
Knowledge base plays an central role in the knowledge support
Transforming reports into actionable knowledge
29
Knowledge Support
Identify similar cases based on query
Web M&M (PSNet)
Patient Safety Organization (PSO) data
Data from home-grown system
Provide solutions and suggestions
30
Prototype
31
Innovative Design
Current Frames
Reports are stored entry by entry
Reporters learn nothing
No feedback for systems
Proposed Frames
Reports are annotated on the same feature tree
Provide solutions for reporters
The system can learn from user feedback and preferences
v
Feedback / Preferences
Knowledge Support (e.g., Solutions)
32
Identifying Relevant Cases
33
Exploring Event Connections
PSE Space
Topic Space
?
Topic Model
34
Developing a PSE Knowledge Base
Topics
Reports
Solutions
A PSE Knowledge Base
35
Acknowledgement
Funding
Agency for Healthcare Research & Quality, 1R01HS022895
UTHealth Innovation in Cancer Prevention Research Training Program
(Cancer Prevention and Research Institute of Texas grant RP#160015)
University of Texas System Grants Program #156374
Current lab
members
Hong Kang, PhD
Pei-Yin Yang, MS
Alumni
Chen Liang, PhD
Ju Wang, PhD
Sicheng Zhou, MS
Bin Yao, MD, MS
Qi Miao, BS
Hsing-yi Song, MD, MS
Xinshuo Wu, MD, MS
Swananda Pandit, MS
Lei Hua, PhD
James Richardson, MS
Zhijian Luan, MS
Yanyan Shen, MHA
Rajitha Gopidi, MHA
Dan Wang, PhD
Mathew Koelling, MHA
CPRIT Summer
intern
Cindy Songting Wu
Frank Wang
Elisa Ali
Melanie Klock
Ethan Wang
Collaborators
Jing Wang, PhD, RN
Nnaemeka Okafor, MD, MS
Hua Xu, PhD
Tina Hilmas RN, BSN
Becky Miller MHA
Amy Vogelsmeier, PhD, RN
36
THANK YOU!
LinkedIn: gongyang@gmail.com
Twitter: gngyng
Please complete online session evaluation
Questions